Systems and Methods for Automated Cloud-Based Analytics for Surveillance Systems with Unmanned Aerial Devices

ABSTRACT

Systems and methods for cloud-based surveillance for a target surveillance area are disclosed. At least two mobile input capture devices (ICDs) are communicatively connected to a cloud-based analytics platform via a data communication device. At least one user device can access to the cloud-based analytics platform. The cloud-based analytics platform automatically analyzes received 2-Dimensional (2D) video and/or image inputs for generating 3-Dimensional (3D) surveillance data and providing 3D display for a target surveillance area. In one embodiment, the at least two mobile ICDs are Unmanned Aerial Vehicles (UAVs).

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is related to and claims priority from the followingU.S. patent Applications. This application is a continuation of U.S.patent application Ser. No. 14/865,684 filed Sep. 25, 2015, which is acontinuation-in-part of U.S. patent application Ser. No. 14/504,132filed Oct. 1, 2014, which it is a continuation-in-part of U.S. patentapplication Ser. No. 14/249,687 filed Apr. 10, 2014, each of which isincorporated herein by reference in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to cloud-based systems and methods forautomated analytics of inputs from remote, distributed devices forsecurity surveillance.

2. Description of the Prior Art

It is known in the prior art to use mobile devices for securitysurveillance, as well as to analyze image and video content forsurveillance purposes. While the prior art discloses individual aspectsas the present invention, very few, if any, teach the ability toauthenticate and analyze captured inputs from un-registereduser-devices. The present invention permits remote servers to acceptcaptured inputs from a variety of mobile devices, authenticate metadatafrom the inputs, and analyze the inputs to provide surveillanceinformation.

The proliferation of wireless, mobile devices having image and videofunctions is widespread and use of these device-functions continues toincrease. Recent years, the development of the unmanned aerial devices(UAVs) boast tremendous potential for surveillance in civilian andmilitary surveillance. Camera equipped UAVs are also a type of mobiledevices. Sporting events, social gatherings, dissident events, andemergency situations are typically captured on a multitude of mobiledevices operated by differing users. Nowhere in the prior art isprovided social surveillance or security system that allows foruploading of these captured inputs, authentication of such inputs, andcloud-based analysis of the inputs in order to provide real- or nearreal-time surveillance of a target environment. Prior art documentsteach that camera and video input devices may be equipped with atime-stamp function that embeds a date and time into an image or videofor later authentication. Also, it is known in the prior art to provideauthentication of users and/or devices through the evaluation ofuploaded content, including stenographic techniques such as digitalfingerprinting and watermarking, or user-verification techniques such aslogin or CAPTCHA technologies and biometric scanning.

Notably, most of the prior art security surveillance systems disclosethe use of fixed devices, rather than the use of mobile devices. Forexample, content-based analytics is widely used in CCTV settings andwhen verifying that digital content has been unaltered or authenticatinga content's source (e.g., copyrighted music, images and videos).Additionally, similar technology has been deployed in military and lawenforcement units, although these technologies typically requirespecialized pre-registered devices, as opposed to incorporatingdistributed, unknown devices.

It is known in the prior art that a video surveillance system can be setup at a location with a local recorder and server besides cameras. Inrecent years, with development of cloud computing and communicationtechnologies, there is a need for users to have access to theirsurveillance systems anywhere anytime with their smart mobile devices.Meanwhile, users need not only basic recording from their surveillancesystems, but also want to get more advanced preventive and proactiveanalytics from their surveillance systems.

Video surveillance systems typically rely on 2-Dimensional (2D) imagesand/or videos. If high-definition 3-Dimensional (3D) images and/orvideos can be generated for surveillance, the security surveillancesystem could harvest much better information. Camera manufactures havedeveloped 3D cameras in order to produce 3D videos. However, the pricesare much higher than those of regular 2D cameras. For the existingsurveillance systems with 2D cameras, it is a huge expense to update to3D cameras in order to get 3D surveillance.

Thus there is a need for a cloud-based analytics platform, which notonly provides users access anyway anytime via a network-connecteddevice, but also generating 3D images and/or videos based on regular 2Dinput data from cameras, especially from mobile devices, and providing3D analytics.

By way of example, prior art documents include:

U.S. Pat. No. 6,842,674 for “Methods and apparatus for decision makingof system of mobile robotic vehicles” by inventor Neal Solomon, filedApr. 22, 2003, describes a swarm weapon system. Decision-makingprocesses are described for the organization of mobile robotic vehicles(MRVs). After MRV drone sensor data is provided to a lead MRV, aninitial decision is made by the lead MRV as a result of the selection ofa winning simulation that provides the best opportunity for success of amission. Once this decision is made, actions are organized for the MRVs,which provide feedback for the continuation of the process until themission is completed.

U.S. Pat. No. 8,903,551 for “Sensor-swarm environmental event detection”by inventor Wael El-Essawy et al. filed Aug. 23, 2011, describes datacenter environmental sensing by a measurement system that detectsenvironmental events from inputs received from a plurality of movablesensors. The sensors are moved in response to detection of an event to aregion of the data center associated with the event, providing increasedspatial resolution of the measurement in the region of the event. Eventssuch as leakage between hot and cold aisles of a data center can bedetected by the system, which may use one or more movable devices thatcan be moved around, between and over equipment, to carry multiplesensors toward the source of the event, providing both diagnostic anddetailed environmental information.

U.S. Publication 2004/0030571 for “System, method and apparatus forautomated collective mobile robotic vehicles used in remote sensingsurveillance” by inventor Neal Solomon filed April 2003, describes asystem for remote sensing surveillance by configuring groups ofautomated mobile robotic vehicles (MRVs). The collection of MRVs havesensors that feed information to a central network node. As the MRVsinteract with the environment, the mobile network adapts in real time byreconfiguring its spatial positions. This system, and the methods andapparatus involved therein, are applied to reconnaissance missions inorder to collect information in remote hostile environments.

U.S. Publication 2004/0143602 for “Apparatus, system and method forautomated and adaptive digital image/video surveillance for events andconfigurations using a rich multimedia relational database” by inventorAntonio Ruiz et al. filed Oct. 17, 2003, describes an automated andadaptive digital image/video and/or sensor surveillance system. Thesystem is provided in a massively and pervasively deployed sensor/imagesurveillance environment using virtual configuration perimeters for allthe subsystems and processes which allow triggered events to beautomatically captured by virtual event perimeters in environments whereunattended operation and automatic support needs to be provided forreal-time event analysis, automatic event tracking, or for storage andretrieval of sensory or visual event information within the scope of thelarge scale spatio-temporal domain of a target surveillance environment.All operations are performed in the framework of the captured data,information, and knowledge derived through fusion operations andcaptured in a relational surveillance database subsystem. Theinformation collected and derived knowledge may be used to dynamicallycreate new virtual event perimeters and new virtual configurationperimeters to enable the system to learn and adapt to events as theytake place.

U.S. Publication 2013/0289858 for “Method for controlling andcommunicating with a swarm of autonomous vehicles using one-touch orone-click gestures from a mobile platform” by inventor Alain AnthonyMangiat et al. filed Apr. 25, 2012, describes a method for controlling aswarm of autonomous vehicles to perform a multitude of tasks usingeither a one touch or a single gesture/action command. These commandsmay include sending the swarm on an escort mission, protecting a convoy,distributed surveillance, search and rescue, returning to a base, orgeneral travel to a point as a swarm. A gesture to initiate a commandmay include a simple touch of a button, drawing a shape on the screen, avoice command, shaking the unit, or pressing a physical button on orattached to the mobile platform.

U.S. Pat. No. 7,259,778 for “Method and apparatus for placing sensorsusing 3D models” by inventor Aydin Arpa et al. filed Feb. 13, 2004,describes method and apparatus for dynamically placing sensors in a 3Dmodel is provided. Specifically, in one embodiment, the method selects a3D model and a sensor for placement into the 3D model. The methodrenders the sensor and the 3D model in accordance with sensor parametersassociated with the sensor and parameters desired by a user. Inaddition, the method determines whether an occlusion to the sensor ispresent.

U.S. Pat. No. 7,675,520 for “System, method and computer program forcreating two dimensional (2D) or three dimensional (3D) computeranimation from video” by inventor Will Gee et al. filed Dec. 7, 2006,describes System, method and computer program for creating twodimensional (2D) or three dimensional (3D) computer animation fromvideo. In an exemplary embodiment of the present invention a system,method and computer program product for creating at least a twodimensional or three dimensional (3D) datastream from a video withmoving objects is disclosed. In an exemplary embodiment of the presentinvention, a method of creating animated objects in 2D or 3D from video,may include: receiving video information which may include a pluralityof frames of digital video; receiving and adding metadata to the videoinformation, the metadata relating to at least one object in motion inthe digital video; and interpreting the metadata and the videoinformation and generating a datastream in at least 2D. In an exemplaryembodiment, 2D, 3D or more dimensional data may be used to provide ananimation of the event of which the video was made. In an exemplaryembodiment, a 2D or 3D gametracker, or play reviewer may be providedallowing animation of motion events captured in the video.

U.S. Pat. No. 7,944,454 for “System and method for user monitoringinterface of 3-D video streams from multiple cameras” by inventorHanning Zhou, et al. filed Sep. 7, 2005, describes a user navigationinterface that allows a user to monitor/navigate video streams capturedfrom multiple cameras. It integrates video streams from multiple cameraswith the semantic layout into a 3-D immersive environment and rendersthe video streams in multiple displays on a user navigation interface.It conveys the spatial distribution of the cameras as well as theirfields of view and allows a user to navigate freely or switch amongpreset views. This description is not intended to be a completedescription of, or limit the scope of, the invention. Other features,aspects, and objects of the invention can be obtained from a review ofthe specification, the figures, and the claims.

U.S. Pat. No. 8,284,254 for “Methods and apparatus for a wide areacoordinated surveillance system” by John Frederick Romanowich, et al.filed Aug. 11, 2005, describes a coordinated surveillance system. Thecoordinated surveillance system uses a larger number of fixed lowresolution detection smart camera devices and a smaller number ofpan/tilt/zoom controllable high resolution tracking smart cameradevices. The set of detection cameras provide overall continuouscoverage of the surveillance region, while the tracking cameras providelocalized high resolution on demand. Each monitor camera device performsinitial detection and determines approximate GPS location of a movingtarget in its field of view. A control system coordinates detection andtracking camera operation. A selected tracking camera is controlled tofocus in on, confirm detection, and track a target. Based on a verifieddetection, a guard station is alerted and compressed camera video isforwarded to the guard station from the camera(s). The guard station candirect a patrol guard to the target using GPS coordinates and a sitemap.

U.S. Pat. No. 8,721,197 for “Image device, surveillance camera, and maskmethod of camera screen” by inventor Hiroyuki Miyahara, et al. filedAug. 10, 2012, describes a microcomputer. In a microcomputer included inan image device, a mask 2D 3D converting section expresses coordinatesof a 2-dimensional image plane defined by an imaging element having arectangular contour in a 3-dimensional coordinate system. The imageplane is positioned in the state that a focal length corresponding to azoom position is adopted as a Z coordinate value of the image plane inthe 3-dimensional coordinate system. A mask display position calculatingsection 165 calculates a 2-dimensional position of a mask on a camerascreen by utilizing a similarity of the size of the image plane and thesize of the camera screen when a position of a mask on the image planein the 3-dimensional coordinate system after PAN, TILT rotations and azooming is converted into the 2-dimensional position of the mask on thecamera screen.

U.S. Publication 2013/0141543 for “Intelligent image surveillance systemusing network camera and method therefor” by inventor Sung Hoon Choi, etal. filed May 23, 2012, describes an intelligent control system. Theintelligent control system according to an exemplary embodiment of thepresent disclosure includes a plurality of network cameras to photographa surveillance area; an image gate unit to perform image processing ofimage data, which is input from the plurality of network cameras,according to a specification that is requested by a user; a smart imageproviding unit to convert a plurality of image streams, which are imageprocessed by the image gate unit, to a single image stream; and an imagedisplay unit to generate a three-dimensional (3D) image by segmenting,into a plurality of images, the single image stream that is input fromthe smart image providing unit and by disposing the segmented images oncorresponding positions on a 3D modeling.

U.S. Publication 2014/0192159 for “Camera registration and videointegration in 3d geometry model” by inventor Henry Chen, et al. filedJun. 14, 2011, describes apparatus, systems, and methods to receive areal image or real images of a coverage area of a surveillance camera.Building Information Model (BIM) data associated with the coverage areamay be received. A virtual image may be generated using the BIM data.The virtual image may include at least one three-dimensional (3-D)graphics that substantially corresponds to the real image. The virtualimage may be mapped with the real image. Then, the surveillance cameramay be registered in a BIM coordination system using an outcome of themapping.

U.S. Publication 2014/0333615 for “Method For Reconstructing 3D ScenesFrom 2D Images” by inventor Srikumar Ramalingam, et al. filed May 11,2013, describes a method reconstructing at three-dimensional (3D)real-world scene from a single two-dimensional (2D) image by identifyingjunctions satisfying geometric constraint of the scene based onintersecting lines, vanishing points, and vanishing lines that areorthogonal to each other. Possible layouts of the scene are generated bysampling the 2D image according to the junctions. Then, an energyfunction is maximized to select an optimal layout from the possiblelayouts. The energy function use's a conditional random field (CRF)model to evaluate the possible layouts.

U.S. Pat. No. 8,559,914 for “Interactive personal surveillance andsecurity (IPSS) system” by inventor Jones filed Jan. 16, 2009, describesan interactive personal surveillance and security (IPSS) system forusers carrying wireless communication devices. The system allows userscarrying these devices to automatically capture surveillanceinformation, have the information sent to one or more automated andremotely located surveillance (RLS) systems, and establish interactivityfor the verification of determining secure or dangerous environments,encounters, logging events, or other encounters or observations. ThisIPSS is describes to enhance security and surveillance by determining auser's activities, including (a.) the user travel method (car, bus,motorcycle, bike, snow skiing, skate boarding, etc.); (b.) the usermotion (walking, running, climbing, falling, standing, lying down,etc.); and (c.) the user location and the time of day or time allowanceof an activity. When user submits uploaded (or directly sent)surveillance information to the public server, the surveillance videos,images and/or audio includes at least one or more of these searchableareas, location, address, date and time, event name or category, and/orname describing video.

U.S. Pat. No. 8,311,983 for “Correlated media for distributed sources”by inventor Guzik filed Dec. 14, 2009 (related to U.S. Publications2010/0274816, 2011/0018998, 2013/0027552 and 2013/0039542) disclosesmethod embodiments associating an identifier along with correlatingmetadata such as date/timestamp and location. The identifier may then beused to associate data assets that are related to a particular incident.The identifier may be used as a group identifier on a web service orequivalent to promote sharing of related data assets. Additionalmetadata may be provided along with commentary and annotations. The dataassets may be further edited and post processed. Correlation can bebased on multiple metadata values. For example, multiple still photosmight be stored not only with date/time stamp metadata, but also withlocation metadata, possibly from a global positioning satellite (GPS)stamp. A software tool that collects all stored still photos takenwithin a window of time, for example during a security or policeresponse to a crime incident, and close to the scene of a crime, maycombine the photos of the incident into a sequence of pictures withwhich for investigation purposes. Here the correlation is both by timeand location, and the presentation is a non-composite simultaneousdisplay of different data assets. Correlating metadata can be based on aset of custom fields. For example, a set of video clips may be taggedwith an incident name. Consider three field police officers each in adifferent city and in a different time zone, recording videos and takingpictures at exactly at midnight on New Year's Day 2013. As a default, agroup may be identified to include all users with data files with thesame Event ID. A group may also be either a predefined or aself-selecting group, for example a set belonging to a security agency,or a set of all police officers belonging to the homicide division, oreven a set of officers seeking to share data regardless of if they arebellowing to an organized or unorganized group.

U.S. Pat. No. 7,379,879 for “Incident reporting system and method” byinventor Sloo filed Feb. 26, 1999, describes a computer-based method ofcollecting and processing incident reports received from witnesses whoobserve incidents such as criminal acts and legal violations. The methodautomates the collection and processing of the incident reports andautomatically sends the incident reports to the appropriate authority sothat the observed incidents can be acted on in an appropriate manner.For example, a witness may be equipped with a video input system such asa personal surveillance camera and a display. When the witnessencounters an incident such as a suspect committing a crime, the videoinput system would automatically recognize the suspect from the videoinput and could then display records for the suspect on the witness'shand held readout without revealing the suspect's identity. The witnesswould not need to know the identity of the suspect to observe theincident relating to the suspect. Such a system may overcome some of theproblems associated with publicly revealing personal data.

U.S. Publication 2009/0087161 for “Synthesizing a presentation of amultimedia event” by inventors Roberts, et al. filed Sep. 26, 2008,discloses a media synchronization system includes a media ingestionmodule to access a plurality of media clips received from a plurality ofclient devices, a media analysis module to determine a temporal relationbetween a first media clip from the plurality of media clips and asecond media clip from the plurality of media clips, and a contentcreation module to align the first media clip and the second media clipbased on the temporal relation, and to combine the first media clip andthe second media clip to generate the presentation. Each user whosubmits content may be assigned an identity (ID). Users may upload theirmovie clips to an ID assignment server, attaching metadata to the clipsas they upload them, or later as desired. This metadata may, forexample, include the following: Event Name, Subject, Location, Date,Timestamp, Camera ID, and Settings. In some example embodiments,additional processing may be applied as well (e.g., by the recognitionserver and/or the content analysis sub-module). Examples of suchadditional processing may include, but are not limited to, thefollowing: Face, instrument, or other image or sound recognition; Imageanalysis for bulk features like brightness, contrast, color histogram,motion level, edge level, sharpness, etc.; Measurement of (and possiblecompensation for) camera motion and shake.

U.S. Publication 2012/0282884 for “System and method for the emergencyvoice and image e-mail transmitter device” by inventor Sun filed May 5,2011, describes a voice and image e-mail transmitter device with anexternal camera attachment that is designed for emergency andsurveillance purposes is disclosed. The device converts voice signalsand photo images into digital format, which are transmitted to thenearest voice-image message receiving station from where the digitalsignal strings are parsed and converted into voice, image, or videomessage files which are attached to an e-mail and delivered to userpre-defined destination e-mail addresses and a 911 rescue team. Thee-mail also includes the caller's voice and personal information, photoimages of a security threat, device serial number, and a GPS locationmap of the caller's location. When the PSU device is initially used, theuser needs to pre-register personal information and whenever a digitalsignal string is transmitted out from the PSU device it will includethese personal information data plus a time code of the message beingsent, the PSU device's unique serial number, and the GPS generatedlocation code, etc. which will all be imbedded in the PSU e-mail.

U.S. Publication 2012/0262576 for “Method and system for a network ofmultiple live video sources” by inventors Sechrist, et al. filed Mar.15, 2012, discloses a system and a method that operate a network ofmultiple live video sources. In one embodiment, the system includes (i)a device server for communicating with one or more of the video sourceseach providing a video stream; (ii) an application server to allowcontrolled access of the network by qualified web clients; and (iii) astreaming server which, under direction of the application server,routes the video streams from the one or more video sources to thequalified web clients.

Geo-location information and contemporaneous timestamps may be embeddedin the video stream together with a signature of the encoder, providinga mechanism for self-authentication of the video stream. A signaturethat is difficult to falsify (e.g., digitally signed using anidentification code embedded in the hardware of the encoder) providesassurance of the trustworthiness of the geo-location information andtimestamps, thereby establishing reliable time and space records for therecorded events. In general, data included in the database may beroughly classified into three categories: (i) automatically collecteddata; (ii) curated data; and (iii) derivative data. Automaticallycollected data includes, for example, such data as reading fromenvironmental sensors and system operating parameters, which arecollected as a matter of course automatically. Curated data are datathat are collected from examination of the automatically collected dataor from other sources and include, for example, content-basedcategorization of the video streams. For example, detection of asignificant amount of motion at speeds typical of automobiles maysuggest that the content is “traffic.” Derivative data includes any dataresulting from analysis of the automatically collected data, the curateddata, or any combination of such data. For example, the database maymaintain a ranking of video source based on viewership or a surge inviewership over recent time period. Derivative data may be generatedautomatically or upon demand.

None of the prior art provides solutions for cloud-based 3D analyticsfor a target surveillance area as provided by the present invention.

SUMMARY OF THE INVENTION

The present invention relates to virtualized computing orcloud-computing network with input capture devices (ICDs) and userdevices and a cloud-based analytics platform for automatically analyzingreceived video, audio and/or image inputs, generating 3D visual data forproviding social security and/or surveillance for a surveillanceenvironment, surveillance event, and/or surveillance target.

The present invention is directed to systems and methods for cloud-basedsurveillance for a target surveillance area. The cloud-basedsurveillance system comprises at least two mobile ICDs, a datacommunication device, a cloud-based analytics platform having aprocessor and a memory, and at least one user device having a displaywith a user interface. The at least two mobile ICDs are communicativelyconnected to the cloud-based platform via the data communication device.The at least two ICDs has at least one visual sensor and are operable tocapture and transmit visual input data to the cloud-based analyticsplatform. The cloud-based analytics platform is operable to controlformation of the at least two mobile ICDs, receive the input data fromthe at least two ICDs, generate 3D visual representation based on inputdata captured from the at least two ICDs and perform advanced analyticsbased on the input data and/or the generated 3D visual representationfor the target surveillance area. The at least one user device isoperable to communication with the cloud-based analytics platform anddisplay the 3D visual representation of the target surveillance area viathe user interface of the user device.

These and other aspects of the present invention will become apparent tothose skilled in the art after a reading of the following description ofthe preferred embodiment when considered with the drawings, as theysupport the claimed invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an exemplary system consistent with theinvention.

FIG. 2 is a flowchart of a method for providing a cloud-basedsurveillance system of the present invention.

FIG. 3 is a schematic diagram of one embodiment of the invention.

FIG. 4 is a schematic diagram of one embodiment of the invention.

FIG. 5 is a schematic diagram of one embodiment of the invention.

FIG. 6 is a schematic diagram of a cloud-based system of the presentinvention.

FIG. 7 is another schematic diagram of a cloud-based system of thepresent invention.

DETAILED DESCRIPTION

Referring now to the drawings in general, the illustrations are for thepurpose of describing a preferred embodiment of the invention and arenot intended to limit the invention thereto.

The present invention relates to cloud-based surveillance systems andmethods for providing at least one server computer in communication witha network for providing centralized and/or distributed cloud-basedanalytics of inputs captured from remote input capture devices forproviding analyzed inputs that are stored in the cloud-based systemdatabase and accessible remotely and securely for providing security forat least one surveillance environment, surveillance event, and/orsurveillance target. Related secure wired and/or wireless networks andsystems, and methods for using them are disclosed in U.S. Publications2006/0064477 and 2014/0071289, and U.S. Pat. Nos. 7,784,080, 7,719,567,7,954,129, 7,728,871, 7,730,534 and 8,395,664, each of which areincorporated herein by reference in their entirety. The presentinvention also relates to generating 3D surveillance data based on 2Dvisual input for providing more accurate 3D analytics. Related 3Dvisualization systems and methods are disclosed in U.S. Pat. No.8,395,664, which is incorporated herein by reference in its entirety.

In the following description, like reference characters designate likeor corresponding parts throughout the several views. Also in thefollowing description, it is to be understood that such terms as“forward,” “rearward,” “front,” “back,” “right,” “left,” “upwardly,”“downwardly,” and the like are words of convenience and are not to beconstrued as limiting terms. Referring now to the drawings in general,the illustrations are for the purpose of describing a preferredembodiment of the invention and are not intended to limit the inventionthereto.

Although “cloud computing” can generically be applied to any software asa service or to services interfacing through the Internet, in thepresent invention, “cloud-based” computing refers to distributedcomputing among at least one server or more than one server.

The present invention provides a cloud-based surveillance systemincluding: at least two mobile ICDs, a data communication device, acloud-based analytics platform having a processor and a memory, and atleast one user device having a display with a user interface. The atleast two mobile ICDs are communicatively connected to the cloud-basedanalytics platform via the data communication device. The at least twomobile ICDs have visual sensors and are operable to capture and transmitvisual input data to the cloud-based analytics platform. The cloud-basedanalytics platform is operable to control formation of the at least twomobile ICDs in the target surveillance area, receive the visual inputdata from the at least two mobile ICDs, generate 3D visualrepresentation based on visual input data captured from the at least twomobile ICDs, and perform advanced analytics based on the visual inputdata and/or the generated 3D visual representation. The at least oneuser device is operable to communicate with the cloud-based analyticsplatform over network and display the 3D visual representation for thetarget surveillance area via the user interface.

In one embodiment, all the mobile ICDs are UAVs. In other embodiment,there are land robots to coordinate with UAVs in the air, and the landrobots and the UAVs are both mobile ICDs.

The at least two mobile ICDs may communicate with each other and form amesh network. In one embodiment, ICDs communicate with each other toextend the range of the mobile ICDs, so they transmit data to pass downthe line to the data communication device. The ICDs communicate witheach other based on set rules and decide themselves when an issue shouldbe made aware of to the cloud-based analytics platform. By way ofexample, one mobile ICD can alert another ICD if it picks up a fastmoving person who is running towards that camera; if a person should notbe within the range of an ICD, it can then alert the cloud platform.

Alternatively, ICDs can communicate with each other to exchange datathat each ICD receives and then, based on rules that each camera has,act on that data. By way of example, if an ICD detects a person who hasan RFID tag, the ICD can also detect that person's RFID data and compareit to a database to determine if that person has permission to be at acertain location. Furthermore, the system also can track a person'smovement. If a person appears with the incorrect RFID tag or no RFIDtag, then an alarm can be sent to other ICDs and/or the cloud-basedanalytics platform which can in turn communicate with other ICDs.

Mobile Input Capture Devices (ICDs)

In the present invention, the at least two mobile ICDs can be unmannedaerial vehicles (UAVs). An unmanned aerial vehicle (UAV), commonly knownas a drone, is a flying vehicle, either autonomously or controlled byremote control. The UAVs used for surveillance can be fixed wing UAVsand/or quadrotors with different sizes. Each UAV may have varioussensors capturing various types of data. In one embodiment, each UAV hasa video camera installed operable to take images and/or videos. Each UAVis operable to determine its flying altitude and global positioningsatellite (GPS) location. In one embodiment, each UAV have thecapability to stamp time and geolocation information to the visual inputdata before transmitting to a cloud-based analytics platform.

In one embodiment, a swarm of UAVs are deployed for a targetsurveillance area. Each UAV provides communication capability among theswarm and a data communication device on the ground. In one embodiment,the UAVs can change flying speed, altitude, direction, etc. In oneembodiment, the UAVs are operable to hover above a certain area and/orland in a certain area.

A swarm of UAVs are capable of gathering more information than a singlecamera. In combination with advanced analytics, this feature of theswarm can become very powerful. This allows for elimination of blindspot limitations associated with a single camera. Static single camerasystem cannot see behind the obstacles. The determination of objectmovement direction becomes much easier to perceive when inspecting ascene from multiple angles.

Data Communication Device

The at least two mobile ICDs are connected to a cloud-based analyticsplatform via a data communication device via a communication network,which can be a Wide Area Network (WAN), a Local Area Network (LAN), or aPersonal Area Network (PAN). The data communication device can be awireless/wired router, an antenna, or a RF transceiver.

Target Surveillance Area

The present invention can be used for property surveillance, eventsurveillance, inspections of power lines, pipe lines, bridges, oilplatforms, and other applications where it is hard or impossible toinstall fixed cameras or it is difficult or dangerous to access by humanbeings. The target surveillance area can be indoor and/or outdoor. Inone embodiment, the target surveillance area has geofence denoting theproperty line and ownership. UAVs deployed for such a targetsurveillance area are operable to sense the geofence signal, orprogrammed to stay within the defined geofence.

Cloud-Based Analytics Platform

The present invention provides a cloud-computing surveillance systemincluding: at least one server computer having a processor and a memory,constructed and configured in network-based communication with amultiplicity of remote input devices having input capture mechanisms;inputs captured by the remote input devices transmitted within a securemessaging communicated over the network; wherein the inputs arereceived, authenticated, and indexed by the at least one server computerand stored in a corresponding database; wherein the inputs are processedand analyzed based upon at least one profile for a surveillanceenvironment, a surveillance event, and/or a surveillance target, forproviding a near-real-time analysis of the inputs to determine a statusof security. The at least one profile associated with the surveillanceenvironment, surveillance event, and/or surveillance target may includesecurity level (low, medium, high), alert level, time interval forreview for change, authorized remote input device and/or userinformation, and combinations thereof. The status may be selected from:normal, questionable, alert, urgent, disaster, injury, and anydescriptor or indicator of the level and condition of the environment,event, and/or target compared with predetermined conditions.

The system may further include a priority and a profile associated withthe inputs for automatically associating the inputs with thecorresponding surveillance environment, surveillance event, and/orsurveillance target. The profile associated with the inputs may includeuser and/or owner identifier, equipment identifier, communicationsecurity level, and combinations thereof. In one embodiment, the securemessaging includes internet protocol (IP) messaging of data packet(s)including the inputs, and may further include encryption, digitalfingerprinting, watermarking, media hashes, and combinations thereof. Asdescribed in the following detailed description of the invention, theinputs are selected from images, audio, and/or video; more particularly,the input is selected from live streaming video, real-time images and/oraudio, previously recorded video, previously captured images and/oraudio, and combinations thereof. The remote input devices include mobilephones, smart phones, tablet computers, portable computers, mobilecommunication devices, wearable input capture devices, and/or securitycameras. By way of example and not limitation, a wearable input capturedevice may be removable, portable devices such as eyewear (like GoogleGlass), headwear, wristwear, etc.

The analysis is performed by a virtualized or cloud-based computingsystem and provides for remote access of analyzed inputs, and involvesat least one rules engine for transforming individual inputs intoanalyzed content. The analyzed content may include inputs from more thanone remote input device. Additionally, the analyzed content may begenerated by transforming the original inputs by the at least one servercomputer automatically assembling input fragments into an integratedcontent file, and wherein the original input is stored and associatedwith the integrated content file.

In one embodiment of the present invention, the authentication includesconfirmation of GPS location of each of the remote input devicesproviding inputs and matching the GPS location with corresponding atleast one predetermined surveillance environment, surveillance event,and/or surveillance target. Preferably, the analysis includesauthentication of the input device with a device identification, a useridentification, a geographic location, and a time associated with theinput and the predetermined surveillance environment, surveillanceevent, and/or surveillance target.

At the at least one server computer, the authenticated inputs areautomatically tagged, combined, grouped, edited, and analyzed by thecloud-based system according to the predetermined surveillanceenvironment, surveillance event, and/or surveillance target. Also, theinput is verified by authenticating the at least one input device and/orits corresponding user and the input is analyzed to confirm that therehas been no alteration, editing, and/or modification to the input priorto its receipt by the at least one server computer.

The present invention also provides methods for the system described inthe foregoing, including the steps of: providing a cloud-based orvirtualized computing system having at least one server computer with aprocessor and a memory, constructed and configured in network-basedcommunication with a multiplicity of remote input devices having inputcapture mechanisms; receiving by the at least one server computer inputsfrom the remote input devices transmitted within a secure messagingcommunicated over the network; authenticating the inputs; indexing theinputs by the at least one server computer; and storing the inputs in acorresponding database; processing and analyzing the inputs by the atleast one server computer using at least one profile for a surveillanceenvironment, a surveillance event, and/or a surveillance target, forproviding a near-real-time analysis of the inputs to determine a statusof security. Additional steps may include: providing a priority for thesecure messaging; analyzing inputs from more than one remote inputdevice in near real time to provide social security surveillance of thesurveillance environment, surveillance event, and/or surveillancetarget; and/or automatically assembling input fragments into anintegrated content file, and wherein the original input is stored andassociated with the integrated content file. Also, preferably, theauthenticating step includes automatic authentication of the inputdevice and/or its user based upon the combination of a deviceidentification, a user identification, a geographic location, and a timeassociated with the input and the predetermined surveillanceenvironment, surveillance event, and/or surveillance target.

The present invention systems and methods include a social surveillancesystem for providing automated cloud-based analytics that allows foruploading of captured inputs, authentication of the inputs, and analysisof the inputs to provide real-time or near real-time surveillance of asurveillance environment, surveillance event, and/or surveillancetarget. The social surveillance invention includes a combination ofseveral key features including input authentication, time, and automatedcloud-based analytics relating to the inputs and the surveillanceenvironment, surveillance event, and/or surveillance target.

The authentication is provided with device and/or user with locationwherein the input devices provide information including geographiclocation information and/or GPS information to be embedded within imagesand videos and/or included in the messaging from the input devices overthe network to the at least one server computer. Additionally, overlayand other techniques may also be used during upload of content, such as,by way of example and not limitation, Time Difference of Arrival (TDOA),Angle of arrival (AOA), and RF fingerprinting technologies.

Preferably, the input devices are equipped with a time-stamp functionthat embeds a date and time into an image or video for laterauthentication, or their messaging provides a date and time associatedwith the inputs, including images, and/or video.

Additionally, the authentication of users and/or devices through theevaluation of uploaded content, including stenographic techniques suchas digital fingerprinting and watermarking, or user-verificationtechniques such as login or CAPTCHA technologies and biometric scanning.

While some content is considered verified by authenticating a user ordevice, additional analytics may be performed by the cloud-based systemto establish that content has not been modified from its originalsources, such as through the use of media hashes. Additionally, afterreceiving and authenticating multiple sources of information, analyticsmay allow for the inputs to be aggregated, tagged, combined, edited,and/or grouped. Although in the prior art, content-based analytics isused in CCTV settings and when verifying that digital content has beenunaltered or authenticating a content's source (e.g., copyrighted music,images and videos), it has not been used for distributed, cloud-basedsocial surveillance allowing for a multiplicity of inputs from remoteinput devices to at least one server computer for analysis of the inputsbased upon a predetermined surveillance environment, surveillance event,and/or surveillance target, and more particularly for securitysurveillance.

Notably, the present invention does not require specializedpre-registered devices, but instead incorporates distributed, andpotentially unknown devices, so long as the user, time and locationcorrespond to the predetermined surveillance environment, surveillanceevent, and/or surveillance target.

Systems and methods of the present invention provide for a multiplicityof remote input devices, by way of example and not limitation, includingcommercially available devices such as Google glass or glasses orheadwear having input capture mechanisms and mobile communicationcapability, mobile smart phones, cellular phones, tablet computers,gaming devices such as an Xbox Kinect controller, so long as the inputdevice is constructed and configured to capture and share or transmitvideo and/or images associated with location data, direction, etc. andowners/users with the cloud-based surveillance system. The inputinformation is stored on at least one server computer, in a centralizedand/or virtualized central manner, and the input information is indexed,organized, stored, and available for access by authorized users via thenetwork through a website or portal or API. The input device ispreferably registered with the system through an app or softwareapplication associated with the remote or distributed input devices.While preregistration is not required for the inputs to be associatedwith at least one surveillance environment, surveillance event, and/orsurveillance target, all inputs are required to be authenticated by thesystem based upon the input device, the input device user, and/orcorresponding identification and/or association with the surveillanceenvironment, surveillance event, and/or surveillance target. By way ofexample and not limitation, a video input is transmitted by a remoteinput device with an email including the video input as a mediaattachment within the message; the cloud-based system and its at leastone server computer receives the email message, authenticates the emailaddress associated with the device and/or user, and accepts the video.Also the same is provided with MMS or text messaging with video and/oraudio and/or image.

In one embodiment of the present invention, method steps include:providing the system as described hereinabove; providing a softwareapplication operating on a remote input device for capturing at leastone input including an image, a video, and/or an audio input; activatingthe software application; capturing the at least one input including animage, a video, and/or an audio input; automatically and/or manuallyincluding structural and/or descriptive metadata, including but notlimited to unique identifying indicia associated with the input, time,location or geographic information, text and/or audio notationassociated with the input, priority flag or indicator, and combinationsthereof.

Optionally, the software application and/or the remote input deviceautomatically verifies and authenticates the user of the remote inputdevice, for example using biometric authentication such as facialrecognition, fingerprint, etc., and/or using a user identification andpasscode or personal identification number, or other authenticationmechanisms. Preferably, the authentication information is included withthe metadata corresponding to the input(s) and associated therewith as acomposite input, and the software application and/or the remote inputdevice automatically transmits the composite input over the network tothe cloud-based system and the at least one server computer thereon andis saved in at least one database. In preferred embodiments of thepresent invention, a user interface is provided on the remote inputdevice(s) or distributed computer device(s) and their correspondingdisplays to provide secure, authorized access to the composite inputand/or to all inputs associated with predetermined surveillanceenvironment, surveillance event, and/or surveillance target stored inthe cloud database.

Also, preferably, the software application on the remote input deviceprovides an automated sharing feature that provides for single clickselect and activation of media sharing of the selected inputs captured.In one embodiment, the single click select and activation of mediasharing of the selected inputs captured on that remote input deviceprovides for automatic association of the shared media with at least oneemail address corresponding to the user and the remote input device.

Swarm Formation

A formation algorithm is designed to achieve certain formations for aswarm of mobile ICDs to deploy surveillance tasks within a targetsurveillance area. The formation refers to the order and/or shape that aswarm of deployed mobile ICDs form. The position and orientation of eachindividual UAV affects its individual and overall view. Different swarmformation coverage of a target surveillance area may yield a differenttype of panoramic view.

There are two parts in the formation algorithm. One part is to form theinitial formation. In one embodiment, UAVs take off independently ofeach other and one at a time, each UAV takes off toward itscorresponding surveillance area and locks onto it in finite time. Inanother embodiment, all UAVs take off simultaneously towards theircorresponding surveillance area and lock onto them at the same instanceof time. The other part is to form dynamic formation. Dynamic formationis necessary when a blind spot is being hit or the required object isnot visible. Dynamic formation plays a vital role upon identifying anobject of interest. The formation will consider the number of UAVs inthe swarm, the requirement to cover the object of interest from variousangles and then apply a suitable formation to the swarm. Dynamicformations ware essentially objective driven. The objective may varyfrom multi-angle surveillance of a specific object of interest toacquisition of sufficient multi-angle data to construct a 3Drepresentation of a target surveillance area.

3D Analytics

The cloud-based analytics platform for a surveillance system may providestorage for visual input data from the at least two mobile ICDs andperform surveillance analytics based on the input data. In oneembodiment, the at least two mobile ICDs are UAVs equipped with cameras.In one embodiment, video input from the UAVs is discontinuous, as theUAVs are operable to fly or hover for a limited period of time. Thecloud-based analytics platform provides advanced image processing,including 3D visual data generation and panoramic image generation.Multi-angle 2D images are used for generation of 3D images. Advancedimage processing on the cloud-based analytics platform finds matchesbetween these 2D images, and the position of matched elements aretriangulated to obtain missing depth information from these two 2Dimages. A 3D image for that one location can be constructed with thedepth information. Similarly, a 3D video can be constructed based on 2Dinput data for streaming and analytics. Generated 3D images and videoscan be rotated to review from different angles.

Thus, the present invention provides robust, real-time or near-real-timeand easy-to-use surveillance analytics. Compare to 2D analytics, 3Danalytics can reduce false alarms, improve the immersive effect for aphysical security presence, and provide more accurate advanced analyticsfunctions, such as facial recognition, object tracking, people counting,etc.

A panoramic image can be generated by image stitching. Image stitchingis the process of combining several images together to form one singlenew image showing the content of individual images in one singlepicture. Similarly, 3D panoramic video can be generated.

The present 3D analytics provides cross-video surveillance and multipletarget tracking. Each movement trajectory of a tracking target may behighlighted differently. An alert may be generated when a target staysin a zone beyond a preset period of time, when a target passes apredefined line, or when a target satisfies any other preset rule fortriggering an alert. The present 3D cloud-based analytics transformspassive analytics to reactive and preventive.

Visual Representation and Display

A surveillance system for wireless communication between componentsincluding: a base system including at least two wireless ICDs and acloud-based analytics platform and a user device having a display with auser interface, the cloud-based analytics platform being operable totransmit and receive information with the ICDs, the ICDs having at leastone visual sensor and at least one input component for detecting andrecording inputs, a microprocessor, a memory, a transmitter/receiver,all ICD components being constructed and configured in electronicconnection; wherein the ICDs are operable for wirelesscross-communication with each other independent of the cloud-basedanalytics platform for forming a mesh network of ICDs operable toprovide secure surveillance of a target environment.

In one embodiment, the user interface provides a visual representationof captured data in an image format and a contextualized image formatcomprising the visual representation of captured data and coordinatedspatial representation of the image format.

Preferably, the coordinated spatial representation of the image formatincludes a coordinate system to provide a spatial context for thecaptured data, which includes narrow-scope context that is relatedspatially to the immediate surroundings, and/or a geospatial context forthe captured data, including more global or broad scope context that isrelated by GPS or other geographic-based coordinate systems. Thus, thepresent invention provides a 3D geospatial view of the captured data.

In one embodiment, the coordinate system is an overlay for the visualrepresentation of the captured data. In this case, the coordinate systemprovides context without visually depleting or diminishing theinformation provided by the two-dimensional or image-based captured dataand its representation on the user interface.

In another embodiment, the coordinate system creates a 3D view of the 2Dimage by providing relational spatial imaging of the surroundingenvironment or context of the image. Preferably, the 2D image isvisually represented as more linearly than the image itself, with thetarget or key aspects of the captured data and/or image beingsubstantially represented in the same manner as in the 2D image view.The target captured data may be the sensed image or object by theICD(s), depending upon the sensors and related functionality. By way ofexample, the target image may be a person whose presence is detected bymotion sensors on the ICD. In any case, the 2D image may be an imageitself, such as a digital photographic image, a still frame of a videoimage, a rendering of the actual image and/or data captured by theICD(s), and combinations thereof.

In a preferred embodiment, the system is operable to provide comparable2D and 3D images as set forth in the foregoing.

The present invention provides for systems and methods having a 3D modelof a space provides a 3D context for the inputs from the ICDs; inputsfrom the ICDs, including direct cross-communication information,location, settings, environment conditions, and inputs (video, audio,temperature, other sensors, object patterns, movement of a multiplicityof objects and/or people, and analytics related to the objects and/orhuman patterns, including visual patterns, predetermined movements orgestures, facial recognition, and combinations thereof), being visuallyrepresented on a GUI independently and in the 3D context forsimultaneous display of all the info, and analytics based on the info,including activity density within the 3D context based on the inputs,for surveillance and analysis of target environment(s).

The present invention provides for custom analytics that are relevant tothe environment as in the present invention. By way of example, in aretail application, it's not about just tracking an individual who mightbe shoplifting or tampering with goods but the relevance is based onpredetermined events or situations, like build-up of customers atspecific 3D locations (like lines at check-out, lines at customerservice, the deli counter, special advertisement or presentation ofarticles in different location to judge traffic/marketing/presentation,the emergency exit, etc.) wherein specific indications (analytics) wouldresult (indication of need to open another register, notify additionalcustomer service reps., more deli people, success of a promotionalevent/packaging change, etc.). This is an “activity density” or “contentdensity” feature and functionality unique to the present invention.Furthermore, other behavior of humans, including but not limited togestures, actions, changes in actions, patterns of behavior, facialrecognition, age, sex, physical characteristics, and combinationsthereof, are preferably included with the 3D visual representation ofthe inputs and the analysis relating thereto. More preferably, theanalysis and indication of predetermined patterns, activities,movements, speed, etc. are included simultaneously with the video inputsand their 3D contextualization to provide for situational awareness andanalysis automatically based upon the inputs and context thereof.

One aspect of the present invention is to provide systems and methodsfor analytics displays and management for information generated fromvideo surveillance systems, including contextualization and remotereview.

Another aspect of the present invention is to provide systems andmethods for analytics displays and management for information generatedfrom direct cross-communication from independent ICDs, wherein theinformation includes contextualization and remote review of inputs fromthe ICDs, the inputs being directly associated with the ICD(s) thatoriginated them, and settings associated with each of the ICDs andinformation associated with the ICD settings (date, time, environmentconditions, etc.) and the inputs (direct correlation).

Another aspect includes the addition of interactive 3D visualizationremotely through a network on a remote computer having a display and agraphic user interface (GUI) viewable by a remote user. Preferably thisremote user GUI provides a true 3D interface for simultaneouslypresenting input information and additional ICD-based information(including but not limited to ICD identification, position, settings,environment conditions, etc.) and an interactive 3D perspective of theICD and its 3D physical context, thereby providing at least three levelsof analytics and visual input information for multi-level processing ofthe surveillance environment.

A smart mesh network surveillance system and method for providingcommunication between a base system having at least one wireless inputcapture device ICD(s) and other ICD(s), wherein the ICD(s) are capableof smart cross-communication with each other and remote access to theirinputs via a server computer, including the steps of providing this basesystem; at least one user accessing the ICDs and inputs remotely via auser interface through a remote server computer and/or electronic devicecommunicating with it, wherein the captured data is represented visuallyon a user interface or screen views for the user, the screen viewsshowing 2D data and corresponding 3D data of the same input capture withcoordinate overlay to provide a geographic context for the captureddata. The present invention uses the aforementioned systems and methodsfor providing a 3D model of a space provides a 3D context for the inputsfrom the ICDs; inputs from the ICDs, including directcross-communication information, location, settings, environmentconditions, and inputs and analysis thereof, being visually representedon a GUI independently and in the 3D context for simultaneous display ofall the info, and analytics based on the info, including activitydensity within the 3D context based on the inputs, for surveillance andanalysis of target environment(s).

Advantageously, this provides for action or response based on the 3Dcontextualized inputs and the various views, including but not limitedto 3D geospatial overlay and interactivity to shift perspective withinthat 3D context.

Video contextualization is selective adopted by the user, preferablythrough a remote, network-based access. That visualization is functionaland operable to be manipulated by a user to provide a visual perspectivethat optimizes data and information review, without eliminating datacontent provided by the input from the digital video surveillancesystem. By way of example and not limitation, the interactive GUIincludes analytics about the target environment, based upon visualpatterns. In one demonstrative case, this may include visual patternsthat are automatically detected in a predetermined environment, such asa retail space. In this setting, automatic notification of a pattern,such as a grouping of a multiplicity of moving objects, like peoplequeuing at a check-out counter, triggers automatic notification that acorresponding action should be taken, such as opening another check-outline to eliminate the queue quickly. In another example, marketinganalytics may be obtained by visual patterns in a 3D environment, suchas traffic around a display in a retail setting; changing displayconfiguration and positioning and the corresponding change in visualpattern detectable automatically in that environment can be comparedusing the systems and methods of the present invention.

3D Display

A user can access to the cloud-based analytics platform via a userinterface via a user device with a display. The cloud-based analyticsplatform has a cloud account associated with a specific surveillancesystem. The user may receive alerts and/or messages via an authorizeduser device, such as smart phones, tablets, personal computers, laptops,head-mounted displays (HIVID), and other display devices.

The cloud-based analytics platform provides 2D and/or 3D video streamingand storage for the surveillance system. A 3D video for a surveillancetarget area, either generated from 2D visual input data or received from3D cameras, can be viewed via the user interface on a user device with adisplay. The 3D video is streaming in real time or near real time. The3D video may not be continuous. In one embodiment, there is one videofor each of the multiple surveillance locations in a surveillance targetarea, and one panoramic video for the entire surveillance target area.

Highlighted trajectory and contextualized features may be displayed withthe 3D video. In one embodiment, the 3D video may be interactive. Forexample, one target object may be viewed from different angles byrotating the 3D surveillance video with a touch screen or a display withcontrol buttons. A user may zoom in the 3D video for closer look, orzoom out the 3D video for a bigger picture.

In one embodiment, the display on a user's device may be conventional 2Ddisplay, then a user may need to wear 3D glasses for 3D view. In anotherembodiment, the display on a user's device may be operable to haveglasses-free 3D display. In another embodiment, the user device is ahead-mounted display, for example Oculus Rift, for virtual realitydisplay.

3D Playback

The cloud-based analytics platform also provides 3D playback for asurveillance target area. 3D playback provides for users to see whathappened in a certain period of time in the past. A certain period ofvideo may be saved automatically on the platform, for examplesurveillance videos for the past 7 days. To obtain video storage andplayback for more than a certain period of time, a user may set thesettings on the platform and a certain fee may be charged.

3D playback provides another chance to identify any other suspiciousobjects and/or phenomena the users may have omitted, or find usefulinformation between targeted objects, or any other information for anauthorized user may be interested in later.

Communications

The mobile ICDs transmits input data and optionally the decisions withinput data wirelessly (using network protocols such as 802.11, cellphone protocols such as CDMA or GSM, or any other wireless protocol suchas Zigbee, Bluetooth, or internet protocol, or other) to a local datacommunication device on the ground (e.g., a router, a RF transceiver, anantenna) and then to the cloud-based analytics platform via internet.

The camera can optionally transmit the data and the decisions and/or thevideo and audio associated with that data wirelessly using networkprotocols such as 802.11, cell phone protocols such as CDMA or GSM, orany other wireless protocol such as Zigbee, Bluetooth, or other) toanother camera which can take that data and combine it with its own datato make unique decisions based on the combination of the two data sets.Then the camera can send the combined data sets and optionally thedecisions and/or video associated with that data wirelessly or wired toanother camera to make further unique decisions on combined data.

Mobile ICDs in Cloud-Based Surveillance Systems

In a cloud-based surveillance system, at least two mobile ICDs areconnected to a cloud-based analytics platform via a data communicationdevice. a user device can access to the cloud-based analytics platformover network communication. By way of example, the at least two mobileICDs are UAVs.

The mobile ICDs may be pre-registered or not registered with thecloud-based analytics platform for the target surveillance area. Eitherway, the mobile ICDs may transmit the input data to the cloud-basedanalytics platform via a data communication device in a secure message,for example, IP message, text message, email, etc. The cloud-basedanalytics platform will confirm the time as to when the input data iscaptured by the mobile ICDs. In one embodiment, the mobile ICDs areequipped with a time-stamp function that embeds a date and time into animage or video for later authentication, or their messages to thecloud-based analytics platform provide a date and time associated withthe input data. The cloud-based analytics platform will also confirm theGPS location of each of the remote input devices providing inputs andmatch the GPS location with that of the target surveillance area. In oneembodiment, the mobile ICDs are equipped with a locating function thatembed GPS information into images/videos for later authentication, ortheir messages to the cloud-based analytics platform providing locationinformation associated with the input data. For example, the textmessages or emails provides location information with the input data.The cloud-based analytics platform will also confirm there has been noalteration, editing, or modification to the input data prior to itsreceipt by the cloud-based analytics platform.

The cloud-based analytics platform generates the 3D representation basedon the authenticated input data. The input data are grouped according totheir location information and/or time information. In one embodiment,two 2D images for one location from different angles are used togenerate a 3D image. Similarly, a 3D video can be generated. In oneembodiment, the target surveillance are is a stadium, people within thestadium can capture images and/or videos within the stadium with theirmobile devices and transmit to a cloud-based analytics platform. Thecloud-based analytics platform authenticates the images and/or videosfor different spots of the stadium from various mobile devices. A 3Drepresentation is constructed based on the images and/or videos. Thecloud-based analytics platform performs advanced analytics based onauthenticated inputs from various mobile devices and the generated 3Drepresentation, including facial recognition and object detection andetc. The 3D representation can be viewed via a user device, and the 3Drepresentation includes an interactive 3D playback for laterinvestigation or other purposes.

FIG. 1 illustrates a block diagram of an exemplary system 100 consistentwith the invention. As shown in FIG. 1, exemplary system 100 maycomprises two mobile ICDs 101, 102, a data communication device 103, acloud-based analytics platform 104 and a user device 105. Thecloud-based analytics platform 104 is constructed and configured innetwork communication with the two ICDs 101, 102 via the datacommunication device 103. The user device 104 can access to thecloud-based analytics platform 104 over network communication. The twoICDs each have a visual sensor 106, 107, respectively. The cloud-basedanalytics platform 104 has a processor 108 and a memory 109. The userdevice has a display with a user interface 111.

FIG. 2 is a flowchart 200 illustrating a method for providing acloud-based surveillance system in the present invention. The methodcomprises (201) communicatively connecting at least two ICDs and atleast one user device having a display with a user interface to acloud-based analytics platform. The method further comprises (202) theat least two ICDs capturing and transmitting input data to thecloud-based analytics platform. The method further comprises (203) thecloud-based analytics platform receiving and authenticating the inputdata; (204) generating 3D visual representation based on the input datafrom the at least two ICDs; and (205) the cloud-based analytics platformperforming advanced analytics based on the input data and generated 3Dvisual representation. The method further comprises (206) the at leastone user device displaying the 3D visual representation of the targetarea via a user interface over a display.

FIGS. 3-5 illustrate schematic diagrams of different embodiments of thepresent invention; like reference indicators are used throughout themultiple figures for the same or similar elements, as appropriate. FIG.3 shows one embodiment of a cloud-based video surveillance system 300.The embodiment shows a CPU processor and/or server computer 120 innetwork-based communication with at least one database 130 and at leastone geographically redundant database 140. The server computer 120 isconnected to a network 110, a communications (wired and/or wireless)router 180, communications tower 160, and a user device 150 are alsoconnected to the network 110. A user device 170 is connected to thenetwork 110 via the communication tower 160. A user device 190 and twoICDs 310 and 320 are connected to the router 180 in a local area networkvia Wi-Fi wireless 601, cellular wireless 602, or Bluetooth wireless603. Each of the two ICDs may include image capture 610, video capture620, audio capture 630, text and audio note 640, and/or geo-location 650technologies, each technology capable of collecting data for upload tothe network 110 and storage on the databases 130, 140. As the userdevice 190 may also contain identity technologies 920, such as facial,fingerprint and/or retina recognition, both databases 130, 140 mayinclude identity database for validating fingerprints, facialrecognition, and/or retina recognition. User devices 150 and 170, beingany computer, tablet, smartphone, or similar device, permits user accessto the data, video, image, and audio storage on the cloud.

FIG. 4 illustrates another embodiment 400 of a cloud-based videosurveillance system providing for the components shown. A communicationsrouter 180 is connected with the network via communication tower 160.

FIG. 5 illustrates another cloud-based video surveillance system 500with the components shown, including a software application or app on acomputing device having a graphic user interface (GUI) providing for alive viewing area on the device and function buttons, virtual buttons(i.e., touch-activated, near-touch-activated, etc.) of record, notes,and send, associated with input capture devices 190.

Referring now to FIG. 6, a schematic diagram 600 illustrating avirtualized computing network used in of one embodiment of the inventionfor automated systems and methods is shown. As illustrated, componentsof the systems and methods include the following components andsub-components, all constructed and configured for network-basedcommunication, and further including data processing and storage. Asillustrated in FIG. 6, a basic schematic of some of the key componentsof a financial settlement system according to the present invention areshown. The system 600 comprises a server 210 with a processing unit 211.The server 210 is constructed, configured and coupled to enablecommunication over a network 250. The server provides for userinterconnection with the server over the network using a personalcomputer (PC) 240 positioned remotely from the server, the personalcomputer has instructions 247 stored in memory 246. There are othernecessary components in the PC 240, for example, a CPU 244, BUS 242,Input/Output (“I/O”) port 248, and an Output (“O”) port 249.Furthermore, the system is operable for a multiplicity of remotepersonal computers or terminals 260, 270, having operating systems 269,279. For example, a client/server architecture is shown. Alternatively,a user may interconnect through the network 250 using a user device suchas a personal digital assistant (PDA), mobile communication device, suchas by way of example and not limitation, a mobile phone, a cell phone,smart phone, laptop computer, netbook, a terminal, or any othercomputing device suitable for network connection. Also, alternativearchitectures may be used instead of the client/server architecture. Forexample, a PC network, or other suitable architecture may be used. Thenetwork 250 may be the Internet, an intranet, or any other networksuitable for searching, obtaining, and/or using information and/orcommunications. The system of the present invention further includes anoperating system 212 installed and running on the server 210, enablingserver 210 to communicate through network 250 with the remotedistributed user devices. The operating system may be any operatingsystem known in the art that is suitable for network communication asdescribed herein below. Data storage 220 may house an operating system222, memory 224, and programs 226.

Additionally or alternatively to FIG. 6, FIG. 7 is a schematic diagramof an embodiment of the invention illustrating a computer system,generally described as 700, having a network 810 and a plurality ofcomputing devices 820, 830, 840. In one embodiment of the invention, thecomputer system 800 includes a cloud-based network 810 for distributedcommunication via the network's wireless communication antenna 812 andprocessing by a plurality of mobile communication computing devices 830.In another embodiment of the invention, the computer system 800 is avirtualized computing system capable of executing any or all aspects ofsoftware and/or application components presented herein on the computingdevices 820, 830, 840. In certain aspects, the computer system 700 maybe implemented using hardware or a combination of software and hardware,either in a dedicated computing device, or integrated into anotherentity, or distributed across multiple entities or computing devices.

By way of example, and not limitation, the computing devices 820, 830,840 are intended to represent various forms of digital computers andmobile devices, such as a server, blade server, mainframe, mobile phone,a personal digital assistant (PDA), a smart phone, a desktop computer, anetbook computer, a tablet computer, a workstation, a laptop, and othersimilar computing devices. The components shown here, their connectionsand relationships, and their functions, are meant to be exemplary only,and are not meant to limit implementations of the invention describedand/or claimed in this document.

In one embodiment, the user device 820 includes components such as aprocessor 860, a system memory 862 having a random access memory (RAM)864 and a read-only memory (ROM) 866, and a user bus 868 that couplesthe memory 862 to the processor 860. In another embodiment, thecomputing device 830 may additionally include components such as astorage device 890 for storing the operating system 892 and one or moreapplication programs 894, a network interface unit 896, and/or aninput/output controller 898. Each of the components may be coupled toeach other through at least one bus 868. The input/output controller 898may receive and process input from, or provide output to, a number ofother devices 899, including, but not limited to, alphanumeric inputdevices, mice, electronic styluses, display units, touch screens, signalgeneration devices (e.g., speakers) or printers.

By way of example, and not limitation, the processor 860 may be ageneral-purpose microprocessor (e.g., a central processing unit (CPU)),a graphics processing unit (GPU), a microcontroller, a Digital SignalProcessor (DSP), an Application Specific Integrated Circuit (ASIC), aField Programmable Gate Array (FPGA), a Programmable Logic Device (PLD),a controller, a state machine, gated or transistor logic, discretehardware components, or any other suitable entity or combinationsthereof that can perform calculations, process instructions forexecution, and/or other manipulations of information.

In another implementation, shown in FIG. 7, a computing device 840 mayuse multiple processors 860 and/or multiple buses 868, as appropriate,along with multiple memories 862 of multiple types (e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core).

Also, multiple computing devices may be connected, with each deviceproviding portions of the necessary operations (e.g., a server bank, agroup of blade servers, or a multi-processor system). Alternatively,some steps or methods may be performed by circuitry that is specific toa given function.

According to various embodiments, the computer system 700 may operate ina networked environment using logical connections to local and/or remotecomputing devices 820, 830, 840, 850 through a network 810. A computingdevice 830 may connect to a network 810 through a network interface unit896 connected to the bus 868. Computing devices may communicatecommunication media through wired networks, direct-wired connections orwirelessly such as acoustic, RF or infrared through a wirelesscommunication antenna 897 in communication with the network's wirelesscommunication antenna 812 and the network interface unit 896, which mayinclude digital signal processing circuitry when necessary. The networkinterface unit 896 may provide for communications under various modes orprotocols.

In one or more exemplary aspects, the instructions may be implemented inhardware, software, firmware, or any combinations thereof. A computerreadable medium may provide volatile or non-volatile storage for one ormore sets of instructions, such as operating systems, data structures,program modules, applications or other data embodying any one or more ofthe methodologies or functions described herein. The computer readablemedium may include the memory 862, the processor 860, and/or the storagemedia 890 and may be a single medium or multiple media (e.g., acentralized or distributed computer system) that store the one or moresets of instructions 900. Non-transitory computer readable mediaincludes all computer readable media, with the sole exception being atransitory, propagating signal per se. The instructions 900 may furtherbe transmitted or received over the network 810 via the networkinterface unit 896 as communication media, which may include a modulateddata signal such as a carrier wave or other transport mechanism andincludes any delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics changed or set in amanner as to encode information in the signal.

Storage devices 890 and memory 862 include, but are not limited to,volatile and non-volatile media such as cache, RAM, ROM, EPROM, EEPROM,FLASH memory or other solid state memory technology, disks or discs(e.g., digital versatile disks (DVD), HD-DVD, BLU-RAY, compact disc(CD), CD-ROM, floppy disc) or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium that can be used to store the computer readableinstructions and which can be accessed by the computer system 700.

It is also contemplated that the computer system 700 may not include allof the components shown in FIG. 7, may include other components that arenot explicitly shown in FIG. 7, or may utilize an architecturecompletely different than that shown in FIG. 7. The various illustrativelogical blocks, modules, elements, circuits, and algorithms described inconnection with the embodiments disclosed herein may be implemented aselectronic hardware, computer software, or combinations of both. Toclearly illustrate this interchangeability of hardware and software,various illustrative components, blocks, modules, circuits, and stepshave been described above generally in terms of their functionality.Whether such functionality is implemented as hardware or softwaredepends upon the particular application and design constraints imposedon the overall system. Skilled artisans may implement the describedfunctionality in varying ways for each particular application (e.g.,arranged in a different order or partitioned in a different way), butsuch implementation decisions should not be interpreted as causing adeparture from the scope of the present invention.

Certain modifications and improvements will occur to those skilled inthe art upon a reading of the foregoing description. By way of exampleand not limitation, the present invention systems and methods mayfurther include automated web-based searching to identify and analyzesimilar images and/or videos (or content, individuals, objects, andcombinations thereof in the images and/or videos) from social websitesor social media postings to associate, link, supplement and/or matchwith the at least one input authenticated and received by thecloud-based server(s) and corresponding to a surveillance environment, asurveillance event, and/or a surveillance target within a predeterminedtimeframe. The above-mentioned examples are provided to serve thepurpose of clarifying the aspects of the invention and it will beapparent to one skilled in the art that they do not serve to limit thescope of the invention. All modifications and improvements have beendeleted herein for the sake of conciseness and readability but areproperly within the scope of the present invention.

The invention claimed is:
 1. A cloud-based surveillance system for atarget surveillance area, comprising: at least two mobile input capturedevices (ICDs), a data communication device, and a cloud-based analyticsplatform; wherein the at least two mobile ICDs are communicativelyconnected to the cloud-based analytics platform via the datacommunication device. wherein the at least two mobile ICDs are operableto capture and transmit visual input data to the cloud-based analyticsplatform; wherein the cloud-based analytics platform is operable to:control formation of the at least two mobile ICDs in the targetsurveillance area; receive the visual input data from the at least twomobile ICDs; generate 3-Dimensional (3D) visual representation based onvisual input data captured from the at least two mobile ICDs; andperform advanced analytics based on the visual input data and/or thegenerated 3D visual representation.
 2. The system of claim 1, whereinthe at least two mobile ICDs are unmanned aerial vehicles (UAVs).
 3. Thesystem of claim 1, wherein the formation comprises an order and a shapethat the at least two mobile ICDs form.
 4. The system of claim 1,wherein the at least two mobile ICDs are operable to communicate witheach other for coordination.
 5. The system of claim 1, wherein the atleast two mobile ICDs are operable to determine and stamp time andgeolocation information to the visual input data.
 6. The system of claim1, wherein the target surveillance area has geofence, and wherein the atleast two mobile ICDs operate within the geofence of the targetsurveillance area.
 7. The system of claim 1, wherein the visual inputdata is discontinuous.
 8. The system of claim 1, wherein the cloud-basedanalytics platform is operable to authenticate, tag, combine, group, andedit the received visual input data.
 9. The system of claim 1, whereinthe cloud-based analytics platform is further operable to provide datastorage, wherein a time period of storage is selectable on thecloud-based analytics platform.
 10. The system of claim 1, furthercomprising at least one user device communicatively connected to thecloud-based analytics platform, and wherein the at least one user deviceis operable to display the 3D visual representation of the targetsurveillance area.
 11. A method of cloud-based surveillance for a targetsurveillance area, comprising: communicatively connecting at least twomobile input capture devices (ICDs) to a cloud-based analytics platformvia a data communication device; the cloud-based analytics platformsetting up the formation of the at least two mobile ICDs in the targetsurveillance area; the at least two mobile ICDs capturing andtransmitting visual input data to the cloud-based analytics platform;the cloud-based analytics platform generating 3-Dimensional (3D) visualrepresentation based on visual input data from the at least two mobileICDs; and the cloud-based analytics platform performing advancedanalytics based on the visual input data and/or the generated 3D visualrepresentation.
 12. The method of claim 11, wherein the at least twomobile ICDs are unmanned aerial vehicles (UAVs).
 13. The method of claim11, wherein the formation comprises an order and a shape that the atleast two mobile ICDs form.
 14. The method of claim 11, furthercomprising the at least two mobile ICDs communicating with each otherfor coordination.
 15. The method of claim 11, further comprising the atleast two mobile ICDs determining and stamping time and geolocationinformation to the visual input data.
 16. The method of claim 11,wherein the target surveillance area has geofence, wherein the at leasttwo mobile ICDs operate within the geofence of the target surveillancearea.
 17. The method of claim 11, wherein the visual input data isdiscontinuous.
 18. The method of claim 11, further comprising thecloud-based analytics platform tagging, combining, grouping, and editingthe received visual input data.
 19. The method of claim 11, furthercomprising the cloud-based analytics platform providing data storage,wherein a time period of storage is selectable on the cloud-basedanalytics platform.
 20. The method of claim 11, further comprising atleast one user device communicating with the cloud-based analyticsplatform and displaying the 3D visual representation of the targetsurveillance area.